## Changes
<!-- Summary of your changes that are easy to understand. Add
screenshots when necessary -->
Added documentation for summary statistics

### Linked issues
<!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes,
fixed, resolve, resolves, resolved. See
https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword
-->

Resolves #514

### Tests
<!-- How is this tested? Please see the checklist below and also
describe any other relevant tests -->

- [ ] manually tested
- [ ] added unit tests
- [ ] added integration tests
- [ ] added end-to-end tests

---------

Co-authored-by: Marcin Wojtyczka <marcin.wojtyczka@databricks.com>

* Added quality checker and end to end workflows ([#519](#519)). This release introduces no-code solution for applying checks. The following workflows were added: quality-checker (apply checks and save results to tables) and end-to-end (e2e) workflows (profile input data, generate quality checks, apply the checks, save results to tables). The workflows enable quality checking for data at-rest without the need for code-level integration. It supports reference data for checks using tables (e.g., required by foreign key or compare datasets checks) as well as custom python check functions (mapping of custom check funciton to the module path in the workspace or Unity Catalog volume containing the function definition). The workflows handle one run config for each job run. Future release will introduce functionality to execute this across multiple tables. In addition, CLI commands have been added to execute the workflows. Additionaly, DQX workflows are configured now to execute using serverless clusters, with an option to use standards clusters as well. InstallationChecksStorageHandler now support absolute workspace path locations.
* Added built-in row-level check for PII detection ([#486](#486)). Introduced a new built-in check for Personally Identifiable Information (PII) detection, which utilizes the Presidio framework and can be configured using various parameters, such as NLP entity recognition configuration. This check can be defined using the `does_not_contain_pii` check function and can be customized to suit specific use cases. The check requires `pii` extras to be installed: `pip install databricks-labs-dqx[pii]`. Furthermore, a new enum class `NLPEngineConfig` has been introduced to define various NLP engine configurations for PII detection. Overall, these updates aim to provide more robust and customizable quality checking capabilities for detecting PII data.
* Added equality row-level checks ([#535](#535)). Two new row-level checks, `is_equal_to` and `is_not_equal_to`, have been introduced to enable equality checks on column values, allowing users to verify whether the values in a specified column are equal to or not equal to a given value, which can be a numeric literal, column expression, string literal, date literal, or timestamp literal.
* Added demo for Spark Structured Streaming ([#518](#518)). Added demo to showcase usage of DQX with Spark Structured Streaming for in-transit data quality checking. The demo is available as Databricks notebook, and can be run on any Databricks workspace.
* Added clarification to profiler summary statistics ([#523](#523)). Added new section on understanding summary statistics, which explains how these statistics are computed on a sampled subset of the data and provides a reference for the various summary statistics fields.
* Fixed rounding datetimes in the checks generator ([#517](#517)). The generator has been enhanced to correctly handle midnight values when rounding "up", ensuring that datetime values already at midnight remain unchanged, whereas previously they were rounded to the next day.
* Added API Docs ([#520](#520)). The DQX API documentation is generated automatically using docstrings. As part of this change the library's documentation has been updated to follow Google style.
* Improved test automation by adding end-to-end test for the asset bundles demo ([#533](#533)).

BREAKING CHANGES!

* `ExtraParams` was moved from `databricks.labs.dqx.rule` module to `databricks.labs.dqx.config`

* Added quality checker and end to end workflows
([#519](#519)). This release
introduces no-code solution for applying checks. The following workflows
were added: quality-checker (apply checks and save results to tables)
and end-to-end (e2e) workflows (profile input data, generate quality
checks, apply the checks, save results to tables). The workflows enable
quality checking for data at-rest without the need for code-level
integration. It supports reference data for checks using tables (e.g.,
required by foreign key or compare datasets checks) as well as custom
python check functions (mapping of custom check funciton to the module
path in the workspace or Unity Catalog volume containing the function
definition). The workflows handle one run config for each job
run. Future release will introduce functionality to execute this across
multiple tables. In addition, CLI commands have been added to execute
the workflows. Additionaly, DQX workflows are configured now to execute
using serverless clusters, with an option to use standards clusters as
well. InstallationChecksStorageHandler now support absolute workspace
path locations.
* Added built-in row-level check for PII detection
([#486](#486)). Introduced a
new built-in check for Personally Identifiable Information (PII)
detection, which utilizes the Presidio framework and can be configured
using various parameters, such as NLP entity recognition configuration.
This check can be defined using the `does_not_contain_pii` check
function and can be customized to suit specific use cases. The check
requires `pii` extras to be installed: `pip install
databricks-labs-dqx[pii]`. Furthermore, a new enum class
`NLPEngineConfig` has been introduced to define various NLP engine
configurations for PII detection. Overall, these updates aim to provide
more robust and customizable quality checking capabilities for detecting
PII data.
* Added equality row-level checks
([#535](#535)). Two new
row-level checks, `is_equal_to` and `is_not_equal_to`, have been
introduced to enable equality checks on column values, allowing users to
verify whether the values in a specified column are equal to or not
equal to a given value, which can be a numeric literal, column
expression, string literal, date literal, or timestamp literal.
* Added demo for Spark Structured Streaming
([#518](#518)). Added demo
to showcase usage of DQX with Spark Structured Streaming for in-transit
data quality checking. The demo is available as Databricks notebook, and
can be run on any Databricks workspace.
* Added clarification to profiler summary statistics
([#523](#523)). Added new
section on understanding summary statistics, which explains how these
statistics are computed on a sampled subset of the data and provides a
reference for the various summary statistics fields.
* Fixed rounding datetimes in the checks generator
([#517](#517)). The
generator has been enhanced to correctly handle midnight values when
rounding "up", ensuring that datetime values already at midnight remain
unchanged, whereas previously they were rounded to the next day.
* Added API Docs
([#520](#520)). The DQX API
documentation is generated automatically using docstrings. As part of
this change the library's documentation has been updated to follow
Google style.
* Improved test automation by adding end-to-end test for the asset
bundles demo ([#533](#533)).

BREAKING CHANGES!

* `ExtraParams` was moved from `databricks.labs.dqx.rule`
module to `databricks.labs.dqx.config`

## 0.9.1

* Added quality checker and end to end workflows ([#519](#519)). This release introduces no-code solution for applying checks. The following workflows were added: quality-checker (apply checks and save results to tables) and end-to-end (e2e) workflows (profile input data, generate quality checks, apply the checks, save results to tables). The workflows enable quality checking for data at-rest without the need for code-level integration. It supports reference data for checks using tables (e.g., required by foreign key or compare datasets checks) as well as custom python check functions (mapping of custom check funciton to the module path in the workspace or Unity Catalog volume containing the function definition). The workflows handle one run config for each job run. Future release will introduce functionality to execute this across multiple tables. In addition, CLI commands have been added to execute the workflows. Additionaly, DQX workflows are configured now to execute using serverless clusters, with an option to use standards clusters as well. InstallationChecksStorageHandler now support absolute workspace path locations.
* Added built-in row-level check for PII detection ([#486](#486)). Introduced a new built-in check for Personally Identifiable Information (PII) detection, which utilizes the Presidio framework and can be configured using various parameters, such as NLP entity recognition configuration. This check can be defined using the `does_not_contain_pii` check function and can be customized to suit specific use cases. The check requires `pii` extras to be installed: `pip install databricks-labs-dqx[pii]`. Furthermore, a new enum class `NLPEngineConfig` has been introduced to define various NLP engine configurations for PII detection. Overall, these updates aim to provide more robust and customizable quality checking capabilities for detecting PII data.
* Added equality row-level checks ([#535](#535)). Two new row-level checks, `is_equal_to` and `is_not_equal_to`, have been introduced to enable equality checks on column values, allowing users to verify whether the values in a specified column are equal to or not equal to a given value, which can be a numeric literal, column expression, string literal, date literal, or timestamp literal.
* Added demo for Spark Structured Streaming ([#518](#518)). Added demo to showcase usage of DQX with Spark Structured Streaming for in-transit data quality checking. The demo is available as Databricks notebook, and can be run on any Databricks workspace.
* Added clarification to profiler summary statistics ([#523](#523)). Added new section on understanding summary statistics, which explains how these statistics are computed on a sampled subset of the data and provides a reference for the various summary statistics fields.
* Fixed rounding datetimes in the checks generator ([#517](#517)). The generator has been enhanced to correctly handle midnight values when rounding "up", ensuring that datetime values already at midnight remain unchanged, whereas previously they were rounded to the next day.
* Added API Docs ([#520](#520)). The DQX API documentation is generated automatically using docstrings. As part of this change the library's documentation has been updated to follow Google style.
* Improved test automation by adding end-to-end test for the asset bundles demo ([#533](#533)).

BREAKING CHANGES!

* `ExtraParams` was moved from `databricks.labs.dqx.rule` module to `databricks.labs.dqx.config`

The release replaces v0.9.0, which were missing PyPI package and will be
removed.

## 0.9.1

* Added quality checker and end to end workflows
([#519](#519)). This release
introduces no-code solution for applying checks. The following workflows
were added: quality-checker (apply checks and save results to tables)
and end-to-end (e2e) workflows (profile input data, generate quality
checks, apply the checks, save results to tables). The workflows enable
quality checking for data at-rest without the need for code-level
integration. It supports reference data for checks using tables (e.g.,
required by foreign key or compare datasets checks) as well as custom
python check functions (mapping of custom check funciton to the module
path in the workspace or Unity Catalog volume containing the function
definition). The workflows handle one run config for each job
run. Future release will introduce functionality to execute this across
multiple tables. In addition, CLI commands have been added to execute
the workflows. Additionaly, DQX workflows are configured now to execute
using serverless clusters, with an option to use standards clusters as
well. InstallationChecksStorageHandler now support absolute workspace
path locations.
* Added built-in row-level check for PII detection
([#486](#486)). Introduced a
new built-in check for Personally Identifiable Information (PII)
detection, which utilizes the Presidio framework and can be configured
using various parameters, such as NLP entity recognition configuration.
This check can be defined using the `does_not_contain_pii` check
function and can be customized to suit specific use cases. The check
requires `pii` extras to be installed: `pip install
databricks-labs-dqx[pii]`. Furthermore, a new enum class
`NLPEngineConfig` has been introduced to define various NLP engine
configurations for PII detection. Overall, these updates aim to provide
more robust and customizable quality checking capabilities for detecting
PII data.
* Added equality row-level checks
([#535](#535)). Two new
row-level checks, `is_equal_to` and `is_not_equal_to`, have been
introduced to enable equality checks on column values, allowing users to
verify whether the values in a specified column are equal to or not
equal to a given value, which can be a numeric literal, column
expression, string literal, date literal, or timestamp literal.
* Added demo for Spark Structured Streaming
([#518](#518)). Added demo
to showcase usage of DQX with Spark Structured Streaming for in-transit
data quality checking. The demo is available as Databricks notebook, and
can be run on any Databricks workspace.
* Added clarification to profiler summary statistics
([#523](#523)). Added new
section on understanding summary statistics, which explains how these
statistics are computed on a sampled subset of the data and provides a
reference for the various summary statistics fields.
* Fixed rounding datetimes in the checks generator
([#517](#517)). The
generator has been enhanced to correctly handle midnight values when
rounding "up", ensuring that datetime values already at midnight remain
unchanged, whereas previously they were rounded to the next day.
* Added API Docs
([#520](#520)). The DQX API
documentation is generated automatically using docstrings. As part of
this change the library's documentation has been updated to follow
Google style.
* Improved test automation by adding end-to-end test for the asset
bundles demo ([#533](#533)).

BREAKING CHANGES!

* `ExtraParams` was moved from `databricks.labs.dqx.rule`
module to `databricks.labs.dqx.config`